Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PiShrink step to the workflow (#1) #54

Merged
merged 1 commit into from
Feb 6, 2024
Merged

Conversation

crs-k
Copy link
Contributor

@crs-k crs-k commented Feb 6, 2024

I believe this covers it. image is much smaller now, compiled and tested it on the same hardware as my previous pr.

You can see the artifact here: https://github.com/crs-k/pwnagotchi-bookworm/releases/tag/v2.8.12

* Add PiShrink step to the workflow

* Add environment variables for Raspberry Pi 64-bit builder

* Update Makefile command to include debug information

* Add source distribution target to Makefile

* Remove unnecessary provisioner command

* Add armhf architecture support

* Remove armhf architecture provisioning

* Refactor Makefile to remove unnecessary files

* Update Makefile command in publish.yml

* Update pwngrid URL in raspberrypi64.yml

* Add permissions to publish workflow

* Remove unnecessary permissions in publish.yml
@jayofelony jayofelony merged commit eab3316 into jayofelony:dev Feb 6, 2024
@jayofelony
Copy link
Owner

Awesome!

@jayofelony
Copy link
Owner

jayofelony commented Feb 8, 2024

image
I am getting this error, I tried running a new publish workflow on v2.8.2.

@crs-k
Copy link
Contributor Author

crs-k commented Feb 8, 2024

that error typically means that the workflow does not have permissions from your repository. in this case, it is trying to call the github api but the token does not have permissions to write the tag (i believe that is the step its on).

a couple options:

1. go to the repository's settings > Actions > General and make sure you're giving permissions to read/write via workflows:

image

2. explicitly give the workflow permissions in the yml file:

name: Publish

on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Version number'
        required: true

permissions:
  contents: write
  packages: write

jobs:
 
  publish:
    runs-on: ubuntu-latest
    steps:

    - name: Remove unnecessary directories
      run: |
        sudo rm -rf /usr/share/dotnet
        sudo rm -rf /opt/ghc
        sudo rm -rf /usr/local/share/boost
        sudo rm -rf "$AGENT_TOOLSDIRECTORY"

    - name: Check disk space
      run: df -BG

    - name: Checkout code
      uses: actions/checkout@v4

    - name: Validate tag
      id: tag-setter
      run: |
            TAG=${{ github.event.inputs.version }}
            if [[ $TAG =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
            echo "Tag $TAG is valid."
            echo "TAG=$TAG" >> $GITHUB_OUTPUT
            else
            echo "Tag $TAG is not a valid semantic version. Aborting."
            exit 1
            fi

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: 3.9

    - name: Install dependencies
      run: | 
           sudo apt-get update && sudo apt-get install -y libdbus-1-dev curl unzip gettext qemu-utils qemu qemu-user-static binfmt-support
           pip install -r requirements.txt

    - name: Update QEMU
      run: |
            sudo update-binfmts --enable qemu-aarch64
            echo $(ls /usr/bin/qemu-aarch64-static)

    - name: Restart binfmt-support
      run: sudo service binfmt-support restart

    - name: Mount binfmt_misc
      run: |
        if ! grep -qs '/proc/sys/fs/binfmt_misc ' /proc/mounts; then
          echo "Mounting binfmt_misc"
          sudo mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
        fi

    - name: Restart binfmt-support
      run: sudo service binfmt-support restart
        
    - name: Update Languages
      run: make update_langs
    
    - name: Compile Languages
      run: make compile_langs

    - name: Check disk space
      run: df -BG

    - name: Check qemu-user-static package
      run: |
        echo "Checking qemu-user-static package..."
        dpkg -s qemu-user-static && echo "qemu-user-static is installed." || echo "qemu-user-static is NOT installed."
    
    - name: Check binfmt-support service
      run: |
        echo "Checking binfmt-support service..."
        service binfmt-support status && echo "binfmt-support service is running." || echo "binfmt-support service is NOT running."
    
    - name: Check binfmt_misc filesystem
      run: |
        echo "Checking binfmt_misc filesystem..."
        mount | grep binfmt_misc && echo "binfmt_misc is mounted." || echo "binfmt_misc is NOT mounted."
        echo $(ls /proc/sys/fs/binfmt_misc | grep qemu-aarch64)

    - name: Run Makefile
      run: make
      env: 
        PWN_VERSION: ${{ steps.tag-setter.outputs.TAG }}

    - name: PiShrink
      run: |
        wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
        chmod +x pishrink.sh
        sudo mv pishrink.sh /usr/local/bin
        find /home/runner/work/ -type f -name "*.img" -exec sudo pishrink.sh {} \;

    - name: Compress .img files
      run: |
         find /home/runner/work/ -type f -name "*.img" -exec xz --no-warn {} \;
    
    - name: Create tag
      uses: actions/github-script@v7
      with:
        script: |
            const version = "${{ steps.tag-setter.outputs.TAG }}"
            console.log(`Creating tag ${version}`)
            await github.rest.git.createRef({
              owner: context.repo.owner,
              repo: context.repo.repo,
              ref: `refs/tags/${version}`,
              sha: context.sha
            })

    - name: Create Release
      id: create_release
      uses: actions/github-script@v7
      with:
        script: |
          const tag = "${{ steps.tag-setter.outputs.TAG }}"
          console.log(`Creating release with tag: ${tag}`)
          const release = await github.rest.repos.createRelease({
            owner: context.repo.owner,
            repo: context.repo.repo,
            tag_name: tag,
            name: tag,
            draft: false,
            prerelease: true,
            generate_release_notes: true
          })
          console.log(`Created release with id: ${release.data.id}`)
          return release.data.id

    - name: Upload Release Asset
      id: upload-release-asset 
      uses: actions/github-script@v7
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        script: |
          const fs = require('fs');
          const path = require('path');
          const release_id = "${{ steps.create_release.outputs.result }}";
          const asset_content_type = 'application/octet-stream';
          const distDir = '/home/runner/work/';
          
          const uploadFile = async (filePath) => {
            if (fs.lstatSync(filePath).isDirectory()) {
              const files = fs.readdirSync(filePath);
              for (const file of files) {
                await uploadFile(path.join(filePath, file));
              }
            } else {
              // Check if the file has a .xz extension
              if (path.extname(filePath) === '.xz') {
                console.log(`Uploading ${filePath}...`);
      
                const asset_name = path.basename(filePath);
                const asset_size = fs.statSync(filePath).size;
                const asset = fs.createReadStream(filePath);
      
                const response = await github.rest.repos.uploadReleaseAsset({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  release_id: release_id,
                  name: asset_name,
                  data: asset,
                  headers: {
                    'content-type': asset_content_type,
                    'content-length': asset_size
                  }
                });
      
                console.log(`Uploaded ${filePath}: ${response.data.browser_download_url}`);
              }
            }
          }
      
          await uploadFile(distDir);
          
    - name: Update Release
      uses: actions/github-script@v7
      with:
        script: |
          const release_id = "${{ steps.create_release.outputs.result }}"
          console.log(`Updating release with id: ${release_id}`)
          github.rest.repos.updateRelease({
            owner: context.repo.owner,
            repo: context.repo.repo,
            release_id: release_id,
            tag_name: "${{ steps.tag-setter.outputs.TAG }}",
            name: "${{ steps.tag-setter.outputs.TAG }}",
            draft: false,
            prerelease: false
          })

    - name: Save environment variable
      run: echo "${{ steps.tag-setter.outputs.TAG }}"  > env_var.txt

    - name: Upload artifact
      uses: actions/upload-artifact@v4
      with:
        name: env-var
        path: env_var.txt

@jayofelony
Copy link
Owner

RIght that worked, it did not upload the final .img.xz file though. And it wasnt set as latest release.

@crs-k
Copy link
Contributor Author

crs-k commented Feb 9, 2024

RIght that worked, it did not upload the final .img.xz file though. And it wasnt set as latest release.

The workflow takes the input for use throughout, but some of the build process will still look at _version.py, I've tried to update the file based on the input, but it does not succeed.

So, updating the version file before using the workflow will work. Also, I do not create a tag or release prior, the workflow will create the files, then create the tag, then create the release in draft and generate release notes. Once the files are uploaded, it will take the release out of draft and be your latest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants